Predicting and Explaining Automatic Testing Tool Effectiveness

Automatic white-box test generation is a challenging problem. Many existing tools rely on complex code analyses and heuristics. As a result, structural features of an input program may impact tool effectiveness in ways that tool users and designers may not expect or understand. We develop a technique that uses structural program metrics to both predict and explain the test coverage achieved by three automatic test generation tools. We use coverage and structural metrics extracted from 11 software projects to train several decision-tree classifiers. These classifiers can predict high or low coverage with success rates of 82% to 94%. In addition, they show tool users and designers the program structures that impact tool effectiveness.